Empirical estimators for semi-Markov processes

Empirical Bayes Estimation in Nonstationary Markov chains

Estimation procedures for nonstationary Markov chains appear to be relatively sparse. This work introduces empirical  Bayes estimators  for the transition probability  matrix of a finite nonstationary  Markov chain. The data are assumed to be of  a panel study type in which each data set consists of a sequence of observations on N>=2 independent and identically dis...

Efficient estimation for semiparametric semi-Markov processes

We consider semiparametric models of semi-Markov processes with arbitrary state space. Assuming that the process is geometrically ergodic, we characterize efficient estimators, in the sense of Hájek and Le Cam, for arbitrary real-valued smooth functionals of the distribution of the embedded Markov renewal process. We construct efficient estimators of the parameter and of linear functionals of t...

On Minimax Optimal Offline Policy Evaluation

This paper studies the off-policy evaluation problem, where one aims to estimate the value of a target policy based on a sample of observations collected by another policy. We first consider the multi-armed bandit case, establish a minimax risk lower bound, and analyze the risk of two standard estimators. It is shown, and verified in simulation, that one is minimax optimal up to a constant, whi...

The KTH Visit in Semi-Markov Processes

Importance Sampling Using the Semi - Regenerative Method

We discuss using the semi-regenerative method, importance sampling, and stratification to estimate the expected cumulative reward until hitting a fixed set of states for a discrete-time Markov chain on a countable state space. We develop a general theory for this problem and present several central limit theorems for our estimators. We also present some empirical results from applying these tec...